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ABSTRACT: This study analyzes the relationship between students' online work habits and 
academic performance. We utilize data from logs recorded by a course management system 
(CMS) in two courses at a small liberal arts college in the U.S. Both courses required the 
completion of a large number of online assignments. We measure three aspects of students' 
online work habits: timeliness, regularity, and intensity. We find that students with high prior 
GPAs and high grades in the course work on assignments early and more regularly. We also find 
that the regularity of work habits during the first half of the term predicts grade in the course, 
even while controlling for the prior GPA. Overall, however, the marginal predictive power of CMS 
data is rather limited. Still, the fact that high achieving students show vastly different work habits 
from low achieving students supports interventions aimed at improving time-management skills. 

Keywords: Work habits, time management, academic performance, predictive analytics, student 
success, online learning behaviour, course management systems 


1 INTRODUCTION 

The relationship between work habits and academic performance has been studied extensively (e.g., 
Britton & Tesser, 1991; Trueman & Hartley, 1996; Zulauf & Gortner, 1999; MacCann, Fogarty, & Roberts, 
2012). One shortcoming of the existing research is that it relies mostly on self-reported measures of 
work habits (see Claessens, Van Eerde, Rutte, & Roe, 2007). This paper improves on this shortcoming by 
using data from an online course management system (CMS) to measure student work habits. The CMS 
records students' every interaction with the system, including date, time, and what part of the course 
website was accessed. This means that we can obtain an accurate record of what students actually do 
rather than what they say they do. 

We use data from two courses that required the completion of a large number of online assignments 
(online quizzes) throughout the term. These assignments had to be completed within the CMS by a 
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specific deadline. Since the assignments were available only through the CMS, and since the CMS 
records every interaction (including when students first look at the assignment), we were able to form a 
picture of student work habits with respect to these assignments. Specifically, we construct three 
measures of online behaviour: 1) timeliness (when students began working on the assignment), 2) 
regularity (whether they tended to work at the same time of day), and 3) intensity (how many times 
they interacted with the course website). 

These three measures admittedly capture a narrower set of behaviours than those captured in the self- 
reported questionnaires used in the literature. For example, Roberts, Schulze, and Minsky's (2006) 
questionnaire, used in several studies, has 36 items. The Britton and Tesser (1991) questionnaire has 18 
items. Nevertheless, our measures of timeliness and regularity correspond directly to sets of questions 
in both questionnaires. Specifically, both questionnaires ask about meeting deadlines, which 
corresponds to our measure of timeliness. It is safe to say that students who repeatedly begin working 
on an assignment at the last minute are bad at meeting deadlines. Similarly, both questionnaires ask 
about planning and organization. The "planning ahead" subscale in Roberts et al. (2006) "reflects an 
individual's preference for structure and routine." This directly corresponds to our measure of 
regularity. 

Understanding the relationship between work habits and academic performance is important for a 
number of reasons. First, as articulated by MacCann et al. (2012), time management "is a set of habits or 
learnable behaviours that may be acquired through increased knowledge, training, or deliberate 
practice." Thus, establishing a link between work habits and academic performance can be used to 
design interventions to promote success among all students. For example, information on online work 
habits of successful students could be made widely available to other students, thus facilitating 
beneficial peer effects. 

The second reason for a formal analysis of the relationship between online behaviour and academic 
achievement is that a number of CMS systems now offer analytics dashboards to alert faculty of student 
engagement with the course website. The utility of the dashboard depends on how predictive the 
information is, and in what direction. For example, the Blackboard Retention Center includes measures 
of student activity on the website measured as the total number of interactions. Colthorpe, Zimbardi, 
Ainscough, & Anderson (2015) find that activity measured as the total number of interactions is actually 
negatively associated with achievement. Our paper offers an additional data point in the relationship 
between various online behaviours and academic achievement. 

We test two main hypotheses. The first hypothesis is that work habits as measured through the 
interaction with the CMS are related to achievement in the course. Specifically, we expect that students 
who begin working on assignments early, work at regular times of day and with greater intensity will 
achieve higher grades in the course. We control for prior academic achievement by using student GPA 
prior to enrolling in the course. Controlling for prior GPA is important since it has been shown that prior 
GPA is a key determinant of achievement in a course (Romer, 1993). 
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Our second hypothesis is that the interactions with the CMS in the first half of the course can be 
predictive of achievement in the course. Specifically, we test whether or not controlling for prior GPA, 
interactions with the CMS as of the first half of the course predict receiving a grade of C or higher. We 
focus on achieving a grade C or higher because a lower level of achievement normally hampers 
retention and progress towards graduation. 

2 LITERATURE REVIEW 

The use of data from a CMS to understand student learning behaviour is not new. Arnold and Pistilli 
(2010) integrated information from a CMS with other student data, including high school GPA, SAT 
score, and the course gradebook, to develop Course Signals, an early warning system currently used at 
Purdue University. As reported in Arnold and Pistilli (2012), the use of the system is associated with 
higher retention rates. Similarly, Jayaprakash, Moody, Laurfa, Regan, and Baron (2014) use CMS data in 
their early warning system: Open Academic Analytics Initiative. They report higher final grades for at-risk 
students who were subject to early notification compared to a control group. Remarkably, their model 
proved effective across four largely different campuses. While the algorithms used in both Arnold and 
Pistilli (2012) and Jayaprakash et al. (2014) use CMS data, it is not clear what specific behaviours 
recorded in the CMS are predictive of academic success. Our paper looks a bit "under the hood" of the 
data from the CMS not only to create a predictive model, but also to understand which behaviours are 
associated with academic success. 

A number of papers specifically use a CMS to measure behaviour outside of the classroom. Yu and Jo 
(2014) use CMS data on 84 undergraduates in a women's university in South Korea. They find that the 
total amount of studying time in the CMS, the regularity of learning intervals, the number of downloads, 
and interactions with peers in the CMS are all positively related to grades in the course. The authors do 
not control for prior GPA nor do they test whether or not some online behaviour early in the course can 
add predictive power to an early warning system. You (2015) uses a CMS to record the late submission 
of assignments and the delays in weekly scheduled learning in order to measure procrastination. He also 
concludes that there is a negative relationship between academic procrastination and course 
achievement. Finally, Colthorpe et al. (2015) combine student surveys with data from a CMS. Similar to 
our paper, they find that high achievement is associated with early submission of intra-semester 
assignments. 

Our paper also relates to the literature on self-regulated learning. Zimmerman (1989) defines self- 
regulated learning as the degree to which students are active participants in their learning process. 
While working on online quizzes and interacting with the CMS, students are on their own. Students 
make their own decisions about when to begin working on the quiz, or to interact with the CMS. Thus, 
timeliness, regularity, and intensity of their online work habits represent a particular set of self- 
regulated behaviours. Winne and Barker (2013) describe a model of self-regulated learning and outline 
how various data can be used to understand these processes. A number of recent papers, e.g., Kumar et 
al. (2005) and Azevedo et al. (2009) show positive effects of providing students with scaffolds that 
enhance learning. In contrast to these studies, our study does not have a strict experimental design. 
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However, documenting a correlation between timeliness, regularity, intensity, and learning outcomes 
(course grade) can contribute to our understanding of how students learn. + 

While CMS records when students interact with a course website, it does not record other activities that 
students do outside of the classroom. To remedy this, Wang et al. (2014) and Wang, Harari, Hao, Zhou, 
and Campbell (2015) use a mobile sensing application to record student work habits, sociability, sleep 
duration, and mental health. Specifically, these researchers focus on the individual behavioural 
differences between high and low academic performers based on the data of a single class of 48 
students at Dartmouth College. These studies show that behavioural factors such as conversational 
interaction, indoor mobility, activity, class attendance, studying, and partying are significantly correlated 
with academic performance. In other words, students with better grades are more conscientious, study 
more, keep social interactions short in the evening, and have lower stress levels throughout the term. 

3 EXPLORATORY DATA ANALYSIS 

The data originate from two courses taught during the 2014-15 academic year with a combined 
enrollment of 78 students. These courses had a similar structure in that they had classroom meetings on 
Monday, Wednesday, and Friday for 10 weeks. The format of the classroom meeting was a combination 
of lecture and discussion. Both courses were taught by the same instructor. In addition, both courses 
had mostly second-year students, and 60% were male. Both courses required students to complete 
about 25 online quizzes prior to nearly every class. The quizzes counted for 25% of the final grade.* * They 
were multiple-choice questions that required either reading or calculations. The quizzes were normally 
made available 36 hours in advance of the 1 a.m. due date. In addition to the quizzes, the course 
website also included a number of readings, past exams, handouts, etc. All interactions with the website 
were recorded in logs. We merge this logs data with information on student GPAs prior to the term they 
took the course. 

Figure 1 shows the average number of visits to the course website by week of the term. In the top panel, 
we group students by achievement in the course. The lowest achieving group includes students who 
earn D, F, or W. § These students stand out as having visited the course website much less frequently 
than did other students. This is particularly pronounced in the second half of the 10-week term. We see 
a similar pattern when we group students by their prior GPA. Students with prior GPA in the bottom 5% 
interacted with the website less than did other students. 


+ In this sense, our work belongs to the literature on design-based research in educational psychology as described by Sandoval 
and Bell (2004). 

* The other components of the grade were midterm and final exams, final paper, and class participation. The correlation 
coefficient between overall course performance and average performance on quizzes is 0.6. 

§ The grade of W indicates student's withdrawal from the course. Students at this institution can withdraw up to the end of 
week eight of the term. Typically, students facing a poor grade choose to withdraw. 


ISSN 1929-7750 (online). The Journal of Learning Analytics works under a Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 


321 



JOURNAL OF LEARNING ANALYTICS 


S9LAR 


(2016). Do the timeliness, regularity, and intensity of online work habits predict academic performance. Journal of Learning Analytics, 3(3), 
318-330. http://dx.doi.org/10.18608/jla.2016.33.15 


Average number of visits by week of the term and grade 



Course grade 

- A, A- (n=13) 

- B+,B (n=23) 

- B-, C+ (n=21) 

C, C- (n=16) 

D, F, W (n=5) 


Average number of visits by week of the term and prior GPA 



Prior GPA 

- Top 15% (n=12) 

- Next Top 25% (n=18) 

- Middle 30% (n=23) 

Next Bottom 25% (n=19) 
-- Bottom 5% (n=4) 


Figure 1: Average visits to course website by week of the term, course grade, and prior GPA 

Average number of visits by hour of the day and grade 
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Figure 2: Average visits to course website by hour of the day, course grade, and prior GPA 
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Figure 2 shows the average number of visits to the course website by the hour of the day. Once again, 
students who received either D, F, or W in the course, or had prior GPA in the bottom 5%, stand out as 
working mostly late at night, including after midnight, rather than in the afternoon. In contrast, high 
achieving students work in the early afternoon. This is particularly true for students with prior GPA in 
the top 15%. Since all quizzes were due at 1 a.m., this suggests that high achieving students began 
working on the quizzes well in advance, while low achieving students left it until the last minute. 

Finally, we examine the regularity of student work habits. For each student and each quiz, we calculate 
how many hours before 1 a.m. the student started to work on the quiz. In order to capture the degree 
to which a student tends to work at the same time of day, we calculate the standard deviation of the 
hours before 1 a.m. across all the quizzes that student took. We do this for all students and then define 
regularity of work habits as the inverse of the standard deviation of the hours before 1 a.m. This 
measure does not depend on whether a student starts working on the quiz early or late, but only on 
whether he or she would start each quiz at the same time of day throughout the term. Students who 
always start at 3 p.m. would have the same regularity of habits as students who always start at 
midnight. 
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Figure 3: Regularity of Study Habits by Course Grade and Prior GPA 
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Figure 3 shows that students who earned grades of B or higher in the course are somewhat more regular 
in their study habits that students who earned lower grades. The lower panel shows how regularity 
varies by prior GPA. The pattern is somewhat mixed in that students with both high and low prior GPA 
have fairly regular study habits. Perhaps high achieving students always start early and low achieving 
students always start late. To see how intensity, timeliness, and regularity interact with each other in 
determining the course grade, we turn to a regression analysis in the next section. 

4 EXAMINING THE RELATIONSHIP BETWEEN STUDENTS’ ONLINE WORK 
HABITS AND ACHIEVEMENT 

4.1 Relationship between Online Work Habits and Course Grade 

In this section, we test the hypothesis that study habits as measured by students' online behaviour are 
related to their achievement in the course. We summarize online behaviour using three variables: 
timeliness, regularity, and intensity. Timeliness measures the average number of hours before the due 
date that students started to work on a quiz. In Table 1, we see that, on average, students started to 
work on quizzes about 7 hours before the due date. Regularity is the inverse of the standard deviation of 
the number of hours before 1 a.m. Finally, intensity is the average weekly number of interactions with 
the website. This includes opening and submitting quizzes, accessing readings, checking quiz results, 
downloading handouts, etc. For example, if a student opens one quiz, submits results for that quiz and 
downloads one reading during a week, his or her intensity would be 3. On average, students had 30 
interactions with the website each week. The average grade in these two courses was 2.7 (B-) and the 
average prior GPA was 3.1* on a 0-4 scale. 


Table 1: Descriptive Statistics 


Statistic 

N 

Mean 

Median 

St. Dev. 

Min 

Max 

Timeliness 

78 

6.8 

7.0 

2.2 

1.7 

11.1 

Regularity 

78 

0.4 

0.3 

0.1 

0.1 

0.9 

Intensity 

78 

29.8 

27.4 

8.5 

17.0 

57.9 

Course Grade 

76 

2.7 

2.7 

0.8 

0.0 

4.0 

Prior GPA 

76 

3.1 

3.0 

0.5 

1.5 

4.0 


In Table 2, we examine the determinants of the course grade. We drop the two students who withdrew 
from the course in this analysis. In the first three equations, we regress the course grade on the three 
aspects of online behaviour. We see that timeliness has a strong effect on achievement in the course. 
Without controlling for prior GPA, each hour that a student starts to work before the due date is 
associated with 0.116 grade points increase in the course grade. Colthorpe et al. (2015) also find that 
early submission of intra-semester assignments is associated with high academic performance. In 


Course grade is missing for two students who withdrew in week 8 from one of the courses. The prior GPA is missing for two 
students. These are likely to be foreign exchange students for whom prior GPA was not available. 
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addition, we find that the regularity of work habits has a positive and statistically significant effect on 
the course grade while the intensity of the interactions with the course website does not appear to be 
associated with the course grade. 

In the fourth specification, we include prior GPA as a control. We see that the statistical significance (and 
magnitudes) of the coefficients on timeliness and regularity drop dramatically. Both timeliness and 
regularity are no longer significant at the 5% level. This indicates that timeliness and regularity are 
correlated with prior GPA. High achieving students, as measured by their prior GPA, tend to work on 
assignments in a timely and regular manner. It is also clear that prior GPA explains far more of the 
variation in the course grade than any other measure of online behaviour. When we include prior GPA in 
the regression, the R-squared nearly quadruples. 


Table 2: Determinants of Course Grade 



(1) 

Dependent variable: 

(2) 

Course Grade 

(3) 

(4) 

Timeliness 

0.116"* 

*** 

0.176 

*** 

0.177 

0.058* 


(0.040) 

(0.046) 

(0.048) 

(0.033) 

Regularity 


1.680** 

1.680** 

0.802* 



(0.709) 

(0.714) 

(0.474) 

Intensity 



-0.002 

-0.013* 




(0.010) 

(0.007) 

Prior GPA 




1.124*” 





(0.114) 

Intercept 

*** 

1.929 

0.913* 

0.949* 

-1.040** 


(0.287) 

(0.511) 

(0.567) 

(0.423) 

Observations 

76 

76 

76 

74 

R 2 

0.102 

0.166 

0.166 

0.651 

Adjusted R 2 

0.089 

0.143 

0.131 

0.631 


Note: p<0.1; p<0.05; * p<0.01 


In Table 3, we try to shed light on the relationship between prior GPA and the three aspects of online 
behaviour. It is clear that timeliness is strongly associated with prior GPA. For every grade point increase 
in prior GPA, students would start working 1.4 hours earlier. It also appears that the intensity of 
interactions with the course website is higher for high GPA students. In contrast, the relationship 
between regularity and prior GPA is statistically insignificant. While there is a clear relationship between 
online work habits, particularly timeliness, and prior GPA, the R-squared ranges between only 0.06 for 
regularity and 0.12 for timeliness. Thus, much of the variation in online work habits is not explained by 
prior GPA. This suggests that work habits are not entirely predetermined, and may in fact be affected 
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through an intervention. One possibility is that such an intervention may include information on the role 
of work habits in student achievement. 


Table 3: Work Habits and Prior GPA 

Dependent variable: 



Timeliness 

Regularity 

Intensity 


(1) 

(2) 

(3) 

Prior GPA 

1.416*” 

-0.011 

4.050*’ 


(0.439) 

(0.031) 

(1.789) 

Intercept 

2.404* 

0.394*’* 

*** 

17.501 


(1.374) 

(0.096) 

(5.602) 

Observations 

76 

76 

76 

R 2 

0.123 

0.002 

0.065 

Adjusted R 2 

0.112 

-0.012 

0.052 

Note: p<0.1; p<0.05; 

*** 

p<0.01 




4.2 Using Online Work Habits to Predict Retention 

In this section, we examine whether or not online behaviour can be used as an early warning system. 
We want to predict when a student will earn a grade of C-, D, F, or W. These grades are of particular 
concern because they hamper student retention and progress towards graduation. We create a binary 
variable called retention that equals one if a student earned a grade of A through C and equals zero if a 
student ended up with a grade of C-, D, F, or W. Approximately 15% of students earned grades C-, D, F, 
or W in the two courses. For an early warning system to be useful, it should only use data available at 
the time the warning is issued. Therefore, we only use data through the first half of the term, i.e., 
through week five. A warning midway through the course can provide both the students and the 
instructors with sufficient notice to try to improve academic performance. 

Table 4 shows the results of a number of probit regressions where retention is the dependent variable, 
and the three aspects of online behaviour and the prior GPA are the independent variables. In the first 
three specifications, we see that both timeliness and regularity in the first half of the term predict 
whether a student earns a C-, D, F, or W in the course. In the fourth specification, we control for prior 
GPA and only regularity remains a statistically significant predictor of earning a C-, D, F, or W. Once 
again, we see that the prior GPA is correlated with timeliness: after controlling for the prior GPA, 
timeliness loses its predictive power. In other words, the information contained in timeliness is already 
contained in the prior GPA. However, the regularity of work habits has predictive power even after we 
control for the prior GPA. Thus, including the regularity of interactions with the course website has the 
potential to add value to an early warning system. 
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Table 4: Predicting Retention Using Online Behaviour during the First Half of Course 



Dep. var: Retention=l if grades A through C, 

=0 otherwise 

(1) 

(2) 

(3) 

(4) 

Timeliness 

0.155* 

0.302” 

0.297** 

0.218 


(0.082) 

(0.121) 

(0.125) 

(0.144) 

Regularity 


3.759" 

** 

3.752 

5.880** 



(1.830) 

(1.833) 

(2.825) 

Intensity 



0.003 

-0.008 




(0.020) 

(0.029) 

Prior GPA 




2.018*** 





(0.566) 

Intercept 

0.018 

-2.253* 

-2.306* 

*** 

-7.896 


(0.545) 

(1.255) 

(1.300) 

(2.434) 

Observations 

78 

78 

78 

76 

Log Likelihood 

-31.659 

-28.944 

-28.933 

-19.321 

Akaike Inf. Crit. 

67.318 

63.888 

65.866 

48.641 


Note: p<0.1; p<0.05; * p<0.01 


5 CONCLUSION 

Professors rarely get to observe how their students study. When professors assign homework, they 
normally do not know when students open their books or begin writing. However, the migration of 
homework from pen and paper to the Internet enables professors and researchers to eavesdrop on 
what students are doing outside of the classroom. The purpose of this paper is to find out if knowing 
about student work habits outside of the classroom could be useful in giving feedback and guidance that 
contribute to their success. 

We find that students who earn high grades start working on assignments earlier than those who do 
not. There is also evidence to suggest that students who earn high grades have more regular work 
habits, i.e., they tend to work on each assignment at the same time of the day. In contrast, the intensity 
of work, as measured by the total number of interactions with the website, is not related to 
performance in the course. We find that prior GPA is by far the most reliable predictor of academic 
achievement in a particular course. However, information about online behaviour, particularly the 
regularity of work habits in the first five weeks of the term, has a mild predictive power even after 
controlling for prior GPA. 
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The CMS data offers insights into the drivers of student success. In particular, we believe that the finding 
that high achieving students start working on assignments early is important. It suggests that academic 
achievement is not just a function of intellect, but also a function of discipline and good time 
management. This will come as no surprise to earlier researchers (Britton & Tesser, 1991; Zulauf & 
Gortner 1999; MacCann et al., 2012) who showed this association in student self-reported data. 
However, confirming the association between work habits and achievement using CMS data is valuable. 
It provides further impetus to interventions aimed at improving study skills and good time management. 
In future research, we plan to investigate whether publicizing the timely and regular habits of successful 
students could improve time management of other students. The work by Sacerdote (2011) shows that 
positive peer effects work through students copying each other's work habits. This suggests that 
publicizing habits of successful students could engender positive peer effects. 
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